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GAO 



United States 

General Accounting Office 

Washington, B.C. 20548 



April 20, 1989 

The Honorable John P. Murtha 
Chairman, Subcommittee on Defense 
Committee on Appropriations 
House of Representatives 

Dear Mr. Chairman: 

In response to your predecessor's request and subsequent discussions with your office, we 
assessed how well the Air Force has managed the Space Defense Operations Center 
modernization effort The replacement system will be an integral part of the North American 
Aerospace Defense Command's ability to provide space surveillance and satellite attack 
warning information to United States leaders. We found that the modernization effort has 
been marked by Air Force iraanagement problems, is significantly behind schedule and over 
budget, and has not met established requirements. 

We are sending copies of this report to the Secretary of Defense; the Secretary of the Air 
Forcej the Chairmen, House and Senate Committees on Armed Services; the Chairman, 
Senate Committee on Appropriations; the Director, Office of Management and Budget; and to 
other interested parries. 

Sincerely yours, 



Ralph V. Carlone 

Assistant Comptroller General 



Executive Summary 



Purpose 



The North American Aerospace Defense Command (norad), a binational 
command supported by the United States Space Cbmmandj is responsi- 
ble for notifying United States and Canadian leaders that North 
America is under air, missile, or space attack The former Chairman, 
Subcommittee on Defense, House Committee on Appropriations asked 
gao to evaluate the Air Force's efforts to modernize United States Space 
Command's space surveillance and attack assessment subsystem— the 
Space Defense Operations Center (spadoc). ih response to this request, 
gao primarily evaluated (1) whether Air Force management of this pro- 
gram has been effective and (2) fee difficulties in meeting program 
requirements. 



Background 



Cheyenne Mountain Air Force Station in Colorado Springs, Colorado 
houses the communications and data processing subsystems supporting 
the Integrated Tactical Warning and Attack Assessment (rnv/AA) system 
for the United States Space Command, which supports norad. spadoc is 
intended to be a data processing and communication center that can 
monitor and maintain orbital information on up to 10,000 man-made 
objects in space, provide timely warning of any threat or attack, and 
protect satellites by identifying the need for satellite maneuvers. 

The on-going spadoc acquisition is divided into three evolutionary blocks 
(A, B, and C). Block A, which entered full scale development in 1983, is 
toprovide the hardware and software to automate the assessment of 
foreign activities (such as a nuclear detonation) that might affect U.S. 
satellites. It also is intended to alert national decision makers of actual 
or potential threats or attacks, and plan and coordinate 
countermeasures. 

Block B, which entered full scale development in 1986, before block A 
was completed, is to improve space surveillance by making orbital posi- 
tion predictions on about 400 satellites of particular interest to the 
Department of Defense. Block C, which has not yet been funded, is to 
complete the automated capability needed to consolidate United States 
Space Command's space defense data processing functions. 

The Air Force plans to spend about $437 million, up from its original 
$290 million estimate, to develop spadoc. And although the entire acqui- 
sition was supposed to be installed and operating by June 1988, the Air 
Force now estimates the system will not be completed until fiscal year 
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1994. In September 1988, the Congress directed that all kohad moderni- 
zation programs, including spadoc, be placed under Defense Acquisition 
Board oversight 



Results in Rrfpf ^ e SPAC0C prog*" 3 " 1 na s been marked by management problems, unreal- 

ized expectations, and program delays. The Air Force has invested over 
S235 million in a system that is now more than 4 years behind schedule 
and far from meeting its required operational capability. 

Tliroughout 5 years of development, technical contractors warned the 
Air Force that the program would have difficulty achieving its require- 
ments. The Air Force continued to press forward with the program 
despite these warnings, consistently deferring problem resolution to 
later development phases. 

The Air Force accepted block A in April 1988, even though it did not 
satisfy most requirements for mission performance. The Air Force 
hoped the deficiencies would be resolved in block B. However, in the 
same month, the Air Force disapproved the block B design because it 
was not expected to meet performance requirements. 

At the root of spadoc's technical problems is the Air Force's attempt to 
achieve controlled mode security. 1 Software development tasks designed 
to achieve this form of multilevel security are timeK»nsuraing, techni- 
cally demanding, and still undergoing much research and development 
In spadoc's case, functions such as notifying national decision makers 
that a satellite is under attack take as much as four times longer to com- 
plete than required. 



Principal Findings 



Problems Surfaced Early 
in Block A Development 



Concerns about block A began to surface shortly after development 
began in April 1983. Mitre Corporation, the Air Force's engineering sup- 
port contractor, repeatedly told the Air Force that it was concerned 
about the adequacy of spadoc's security design, about inadequacies in a 
model used by the contractor to predict performance, and about overall 
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software design quality. At critical design review in March 1984, system 
design should have been complete but was not However, the Air Force 
decided to continue development based on the incomplete design 
because it believed it was less risky than delaying development This 
began a continuing pattern of approving key project milestones although 
requirements had not been satisfied and deferring problem resolution. 



Correcting Block A 
Deficiencies Deferred to 

Block B 



The Air Force accepted block A almost 3 years late, but even given this 
extra development time, the system still could not achieve controlled 
mode security and only met 7 of 23 required mission functions stated in 
the contract. The Air Force acknowledged the problems the contractor 
was having meeting the security requirement in September 1984 and 
deferred the security requirement to block B. In March 1988, the Air 
Force deferred meeting specific block A performance requirements to 
block B because the Air Force believed that even with fine tuning the 
system would not perform as required. 



Problems Identified in 
Block A Are Continuing in 
Block B 



When the Air Force began developing block B in June 1986, it knew that 
many of the block A problems could seriously impair block B develop- 
ment Although a different system performance model was being used, 
the Air force was aware of concerns that the new model also could not 
accurately predict system performance. Further, the system design had 
not shown it could meet the performance and controlled mode security 
requirements. This inability of the system design to meet the require- 
ments finally led to the Air Force's disapproval of the block B design in 
April 1988, following critical design review and after almost 2 years of 
development 



The contractor has been allowed to continue developing the block B sys- 
tem, even though the Air Force has not approved the hardware config- 
uration, software approach, or system design. As with- block A, 
continuing on this track only serves to increase program costs and risks 
with no discernable benefit. 
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Recommendations to 
the Secretary of 
Defense 



gao recommends that the Secretary of Defense halt block B development 
until the Air Force has complied with congressional requirements to sub- 
mit NORAS modernization programs, including SPADOC, to Defense Acqui- 
sition Board review. During that review, the Secretary of the Air Force 
should specifically submit to the board: 

Recommendations on the ultimate disposition of the spadoc block A sys- 
tem, including an evaluation of the capabilities and deficiencies of the 
block A system as accepted and recommendations on how block A 
should be used based on careful analysis of costs incurred and benefits 
derived. 

Plans for the future spadoc system including a thorough analysis of the 
requirements for spadoc and the feasibility of satisfying those require- 
ments, in particular controlled mode security. 



Recommendations to 
the Congress 



gao recommends that the Congress withhold further funding for the 
spadoc 4 acquisition until the Defense Acquisition Board has submitted 
and the Secretary of Defense has approved program plans meeting the 
objectives described in gao's above recommendations. 



Agency Comments 



The Department of Defense agreed with most of the information pre- 
sented in the report, but did not concur with several of the findings and 
conclusions and disagreed with gao's recommendations to withhold 
funding and halt development. Defense stated it has effectively man- 
aged the program and adequately assessed the program's risks. How- 
ever, gao's report provides numerous examples where Air Force 
management did not resolve but continually deferred problem resolu- 
tion. Further, gao did not find evidence that the risk involved in achiev- 
ing critical requirements was assessed. Finally, gao's recommendations 
to withhold funding and halt development are designed to minimize 
future cost growth and schedule delays and give the Air Force an oppor- 
tunity to resolve complex technical problems confronting the spadoc 4 
program. Chapter 5 contains gao's evaluation of Defense's comments. 
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Introduction 



The Space Defense Operations Center (spadoc) is a centralized command 
and control facility being developed to support the surveillance of all 
man-made objects in space and to assess attacks on U.S. satellites. 
spadoc is a major component of the United States Space Command's Inte- 
grated Tactical Warning and Attack Assessment (itw/aa) system— 
formerly called TW/AA. The itw/aa system provides the U.S. Space 
Command and the North American Aerospace Defense Command 
(norad) with timely warning and assessments of any attack on North 
America, including attacks on satellites. 

norad is a binational military command consisting of U.S. and Canadian 
personnel. American and Canadian leaders rely on norad to provide sur- 
veillance of North American airspace, and warn of bomber and missile 
attacks, norad is supported by the United States Space Command, a uni- 
fied command made up of three components: Air Force Space Command, 
Naval Space Command, and the United States Army Space Command, 
which oversee U.S. missile warning and space surveillance. The Air 
Force Space Command, as a component of U.S. Space Command, pro- 
vides the majority of ground and space-based systems, equipment, and 
personnel that enable norad and U.S. Space Command to perform their 
missions. The Commander-in-Chief of U.S. Space Command also serves 
as the Commander-in-Chief of norad. 



The ITW/AA System 



The command and control center for the itw/aa system is the Cheyenne 
Mountain Air Force Station in Colorado Springs, Colorado. This facility 
houses data processing and communications equipment supporting the 
warning and assessment missions of U.S. Space Command and norad. 
The itw/aa system, which the Air Force calls a "system of systems," 
consists of air defense, space defense, and missile warning subsystems, 
as well as communication links, systems for correlating information, and 
display terminals. 

Because of evolving mission requirements and threats, the itw/aa sys- 
tem is constantly changing. The current modernization will upgrade the 
existing system, which consists of hardware and software predomi- 
nantly dating from the mid-1970s. Appendix I illustrates the current 
system, which is comprised of five major subsystems: (1) the Communi- 
cation System Segment, (2) Space Defense Command and Control Sys- 
tem, (3) norad Computer System, (4) Mission Essential Back-up System, 
and (5) the Intelligence Data Handling System. Appendix II shows the 
system as it will look when modernized by replacement subsystems, spe- 
cifically with the Communication System Segment Replacement, the 
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Command Center Processing and Display System Replacement, the 
Survivable Communications Integration System, Granite Sentry, and 
spadoc 4. spadoc 4 is the fourth upgrade to the spadoc program designed 
to replace spadoc 3 and the Space Surveillance Center, parts of the Space 
Defense Command and Control System. Although each of these moderni- 
zation programs is required to interact with one or more of the others, 
all the programs are being developed, acquired, and installed separately. 



the ITW/AA System 



How SPADOC Fits Into ^ ne ^ Force "^tiated the spadoc program to be a single command 

center for all command, control, and communications, and data process- 
ing functions for space defense activities. The current system receives 
observations on satellites and other objects in space from radar and 
optical tracking sensors worldwide, processes this information to update 
satellite orbits, and upon request, provides satellite orbit and threat 
information to satellite owners and operators. 1 Currently, much of the 
tracking information on man-made objects in space — from large satel- 
lites to screwdrivers and gloves left in space during space shuttle mis- 
sions—and the information used to protect satellites from threats and 
attacks is processed manually within the Space Surveillance Center. A 
modernized spadoc is intended to automate these functions and provide 
up-to-date satellite states information to other subsystems in Cheyenne 
Mountain, such as U.S. Space Command's Space Command Center, and 
to national decision makers when the need arises. (See app. H.) spadoc is 
being implemented in phases. As discussed below, spadoc phases 1 
through 3 are complete, and phase 4, the current modernization effort, 
is under development spadoc 4 is also being built in phases called spadoc 
4 blocks A, B, and C. Initially, the spadoc 4 program was scheduled for 
completion by June 1988 at a cost of $289.6 million. However, due to 
development problems and schedule delays, the Air Force now antici- 
pates the program will be completed in fiscal year 1994 at a cost of 
$437 million. 



The SPADOC Mission and 
Program Evolution 



The spadoc mission is to monitor space activities, inform decision mak- 
ers of a threat or attack, and help protect satellites. The "monitor and 
inform" mission includes space surveillance, detecting and identifying 
threats to U.S. and allied space systems, and disseminating information 
to key decision makers. The "protect" mission includes assisting U.S. 
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satellite owners and operators to improve the survivability of their 
space systems by identifying and coordinating actions such as satellite 
s to avoid threats. 



The spadoc program began in 1979 and has evolved through three 
phases to date, spadoc 1, completed in 1979, provided a location for 
Sfadoc operations in Cheyenne Mountain, and established formal proce- 
dures and message formats 2 for communicating with satellite owners 
and operators, spadoc 2, completed in 1981, added a manual subsystem 
to manage the data base that provides information on thousands of 
objects in space, spadoc 3, completed in 1983, added dedicated communi- 
cation lines between the system and satellite owners arid operators. The 
current spadoc 3 functions as the command and control center for Ghey- ' 



SPADOC 4, the last spadoc modernization phase, will add data processing 
equipment to (1) computerize the existing, manually maintained, space 
object data base; (2) monitor and assess additional space activities (such 
as lasers and the effects of nuclear detonations); and (.3) expand the 
ability to perform several space defense activities concurrently. Because 
spadoc 4 will not perform all functions currently performed in the Space 
SurvefllanceCenter until block C is complete, the Space Surveillance 
Center and spadoc 4 will operate concurrently using the same space 
object data base until at least fiscal year 1994. 



Organizations 
Involved With 
SPADOC 



U.S. Space Command is the user of the spadoc system, which is being 
acquired by Air Force Systems Command's Electronic Systems Division. 
Air Force Space Command, a component of U.S. Space Command, will 
manage and maintain spadoc and provide U.S. Space Command with per- 
sonnel and equipment to accomplish its space defense mission. Ford 
Aerospace and Communications Corporation (hereafter referred to as 
Ford) is the spadoc prime contractor and International Business 
Machines (bm) is the major hardware subcontractor. Mitre Corporation 
is providing engineering support to Electronic Systems Division for 
spadoc. Logicon, Incorporated is Electronic Systems Division's indepen- 
dent validation and verification contractor for spadoc, and is responsible 
for independently evaluating the system to determine whether it will 
satisfy mission and operational requirements. 
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Objectives, Scope, and 
Methodology 



This report is the second of three reports in response to a request from 
the former Chairman, Subcommittee on Defense, House Committee on 
Appropriations that we assess the Air Force's management and system 
integration of computer modernization programs for korad's Cheyenne 
Mountain Complex. 3 This report assesses the Air Force's progress in 
acquiring blocks A and B of the spadoc 4 program (block C is planned 
but unfunded). Specifically, we evaluated (1) why block A does not meet 
Air Force requirements and is significantly behind schedule, (2) whether 
block A's problems are likely to be corrected in block B, and (3) whether 
Air Force management of the program has been effective. 

To determine why block A does not meet Air Force requirements, we 
reviewed system requirement documents, contracts (including system 
spedficarions), test reports, and other program documentation, and dis- 
cussed these documents and related information with program officials 
from the Air Force's Electronic Systems Division, Air Force Space Com- 
mand, Ford, Mitre Corporation, and Logicon, Incorporated. We also dis- 
cussed the relative importance of spadoc system requirements with Air 
Force Space Command and U.S. Space Command officials- We compared 
test results with system requirements and specifications to determite 
the extent and causes of performance shortfalls, and discussed this 
information with Air Force and contractor officials- We also determined 
the extent and causes of schedule delays by reviewing program docu- 
mentation and discussing this information with program officials- 

To determine whether block B is likely to correct block A's development 
problems, we reviewed system requirements and specifications, system 
design documentation, and other program documentation, and discussed 
the design and changes made or being considered with Air Force and 
contractor officials. We evaluated documentation pertaining to a model 
developed by the contractor to predict the system's performance and 
discussed the model's capabilities and deficiencies with Air Force and 
contractor officials. 

To evaluate Air Force program management effectiveness, we reviewed 
information contained in contract and program files, minutes from pro- 
gram management reviews and working group meetings, and correspon- 
dence between the Air Force and the contractors. We interviewed Air 
Force and contractor officials to clarify the information contained in the 
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documents reviewed, and to obtain more information about Air Force 
and contractor management of the spadoc program. 

We conducted our work at Air Force Headquarters and Air Force Sys- 
tems Command in Washington, D.C.; Air Force Systems Command's Elec- 
tronic Systems Division at Hanscom Air Force Base, Massachusetts; U.S. 
Space Command and Air Force Space Command at Peterson Air Force 
Base, Colorado; Ford Aerospace and Conuhunicatioiis Corporation and 
Logicon, Incorporated at ColoradeSprings, Colorado; and Mitre Corpora- 
tion, Bedford, Massachusetts. Our work was conducted from February 
through September 1988. Information has been updated through 
November 1988. Our work was performed in accordance with generally 
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Chapter 2 



SPADOC 4 Is Not Yet Operational and Is 
Significantly Over Cost and Behind Schedule 



spadoc 4 did not become operational in June 1988 as originally planned 
and is significantly over cost and behind schedule. Block A was accepted 
by the Air Force in April 1988 although it does not meet all contractu- 
ally specified requirements and is not yet operational. 1 Correcting block 
A deficiencies was deferred to block B. Block B has been under develop- 
ment since 1986, but its design was rejected by the Air Force in April 
12S8 because it too was not expected to meet contractually specified 
requirements. A review of another design is scheduled for April 1989- 
While the Air Force has not yet obtained congressional funding for block 
C development, through August 1988 it has spent S235.8 million devel- 
oping blocks A and B. Under the Air Force's best scenario, the program 
will become operational at least 6 years late and S147 million over the 
original budget 



SPADOC Performance 
Expectations 



Trie Air Force has established certain performance requirements (called 
quantitative performance requirements or qprs) as the primary means 
for measuring system performance, qprs specify how quickly the system 
should process incoming data and generate the required output 
messages. According to Air Force Space Command officials, the most 
important of these QPRs are 23 that are designated "mission related." 
These mission related qprs include specifications for system perform- 
ance when: 

identifying and assessing threats from orbital interceptors, nuclear 
detonations, electronic warfare, and lasers; and 
disseminating timely information about those threats to officials and 
organizations in the space community, such as the norad command post 
and satellite owners or operators. 



Security 



The spadoc contract specified that block A operate in "controlled mode," 
or be capable of evolving to this mode of operation. A system operating 
in controlled mode is intended to assure that users cleared at secret, con- 
fidential, or unclassified levels can access only the information to which 
they are entitled, that is, a system operating in controlled mode will not 
permit an unclassified user to access secret data. 



ig thesystem on behalf of 



fully completes operation 



GA0/BITEC89-18 Ojmtj 



Block A Is Not Yet 
Operational 



Although block A has been in development since April 1983, it can not 
perform 14 of 23 required mission functions quickly enough to satisfy 
contractually specified quantitative performance requirements. 2 Some of 
these functions, such as notifying national decision makers that a satel- 
lite is under attack, take as much as four times longer to complete than 
specified. Additionally, the system does not automatically ensure that 
confidential or secret information will not be sent to unauthorized satel- 
lite owners and operators. This function is done manually under the 
existing system and will continue to be done manually with block A. 



The Air Force concluded that these difficulties would have an impact on 
spadog's completion schedule, and would have to be resolved during 
block B development In September 1984, the Air Force deferred meet- 
ing the controlled mode security requirement to block B. In April 1988, 
according to the Vice Commander, Electronic Systems Division, the Air 
Force deferred meeting the quantitative performance requirements to 
block B and accepted the block A system to "get some minimal and mar> 
ginal capability into Air Force Space Command's hands." 

When block A was accepted, 295 other deft jencies and enhancements 
had been identified by the Air Force, JVPtre, and logicon. Ford has 
agreed to correct 90 deficiencies that Electronic Systems Division identi- 
fied as significant and as needing correction before the system can 
become operational. No plans have been made to address the 205 
enhancements because Electronic Systems Division does not consider 
them to be contract requirements. Testing was conducted between Sep- 
tember and December 1988 to verify that the 90 deficiencies were cor- 
rected. Air Force Space Command is supposed to finish operational 
testing to establish block A's operational capability inMarch 1989. The 
Air Force currently anticipates declaring block A operational in April 



Thp Air Fnrrp "RpipHw! Additional space surveillance functions are to be automated in block B. 
■Di ID' rf • ^J^ 1 *^ Block B is to establish a fully automated space object data base that can 
BlOCK B S Design catalog at lei st 10,000 objects, including the 7,000 objects currently 

maintained in the Space Surveillance Center. Block B is to maintain and 
update orbital data and make orbital position predictions on about 400 
satellites of particular interest to the Dejwtrnent of Defense. Block B is 



■edaraifiaj, and tlterefd 



GAO/SMTEC83-18 Orations Cent 



also expected to process up to 100,000 observations of satellite position 
per day, double the 50,000 observations normally processed currently. 

In June 1986, the Air Force modified the spadoc 4 contract to include 
block B, with initial operational capability scheduled for January 1989. 
In December 1987, Ford presented its block B design for Air Force 
approval. The design incorporated proposed design changes to overcome 
block A performance problems. Ford proposed that the computer hard- 
ware be upgraded, and many system requirements be reduced because, 
in Ford's opinion, the contract requirement to concurrently meet the 
qprs and controlled mode security specified in the contract could not be 
met, even with upgraded equipment In April 1988, after almost 2 years 
of development, the Air Force rejected Ford's block B design proposal, 
stating that it was not expected to meet the qprs. The Air Force empha- 
sized to Ford that it is required to deliver a system design that will meet 
all operational requirements. 



In August 1988, Ford agreed to redesign block B in another attempt to 
meet all spadoc operational requirements. Another design review has 
been scheduled for April 1 989 to evaluate Ford's revised block B design 



Rlnrk P T<? Planned hnt SlockC is intended to complete the transition from the existing Space 
oiuut Vj lb r wiuku uul Survemance q^^ system md pr0V ide the automated capability 
Not Yet Funded needed to manage U.S. Space Command's space defense mission. Block C 

is to maintain and update orbital data, and make orbital position predic- 
tions on at least 10,000 potential objects in the space object da$a base- 
When block C is completed, the Space Surveillance Center will he retired 
and its functions will be performed by spadoc 4. 

Under current Air Force schedules for modernizing the itw/aa system, 
block C is to be operational in 1994 to complete the transition from the 
existing system. As of February 1989, the Air Force had not obtained 
congressional funding nor made specific plans to award a contract to 
design and develop block C. 
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SPADOC 4 Is Not Yet Operational and Is 
Significantly Over Cost and Behind Schedule 



SPADOC 4 Acquisition 
Costs Have Risen and 
Milestones Have Been 
Delayed 



The estimated cost to acquire spadoc 4's three blocks has risen from 
$289.6 million in May 1982 to $437 million, as of March 1988, an 
increase of about $147 million. The differences in these estimates, bro- 
ken down by appropriation, are shown in table 2.1. 



Table 2.1: SPADOC 4 Cost Estimates 





Estimate ; 


ssof: 


Appropriation 


May 1S82 


March 1988 


Research. Development, Test and Evaluation 


S1067 


S315.8 


Oilier Procurement 


151.7 


114.0 


Operations and Maintenance 


31-2 


55 


Military Construction • 1.7 


Total 


S289.6 


$437.0 



In February 1989, the Air Force said it was re-estimating spadoc's cost in 
preparation for the fiscal year 1991 budget request Preliminary Air 
Force estimates indicate the cost to complete the spadoc 4 program may 
increase to $446 million. 



In addition to increasing costs, the scheduled completion of spadoc 4's 
three blocks has been delayed significantly. A comparison of the original 
and current completion dates for each block is shown in table 2.2. 



Table 2.2: SPADOC 4 Completion Dates 





Scheduled completion 


SPADOC 4 


As of Hay 1982 


Current 


Block A 


December 1984 


April 19S9 


BtockS 


December 1986 


March 1990 


Block C 


June 1988 


fiscal year 1994 



SpadOC's original cost estimates did not meet the expenditure thresholds 
that would have required Defense-level oversight throughout develop- 
ment. As cost estimates rose above these thresholds early in the spadoc 
contract, Defense and the Air Force did not reconsider the original deci- 
sion to manage the program without formal Defense oversight. In Sep- 
tember 1988, the Congress directed that all of nosad's modernization 
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Chapters 
Significantly 



programs, including spadoc, be consolidated and placed under the over- 
sight of the Defense Acquisition Board. 3 Congress further required that 
the board conduct a management review of the consolidated program 
during fiscal year 1989 and report the results to the Congress. 



3 Makinfl Appropriations for the Department of Defense , House of Representatives Confer* 
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Chapter 3 

Air Force Deferral of Block A Problems 
Resulted in a Marginal System 



From the outset of the block A acquisition, the Air Force was aware that 
achieving some block A requirements would be risky, and because of 
this, the program needed close management oversight However, the Air 
Force did not adequately assess and mitigate early risks and later 
allowed the program to continue without resolving identified problems 
or providing adequate program management During design and devel- 
opment, the Air Force was frequently alerted by its technical support 
contractors of their concerns about the progress of the security design, 
the adequacy of the spadoc performance prediction model, overall soft- 
ware design and quality, and the possibility that the system might not 
meet performance requirements. Nevertheless, the Air Force allowed 
Ford to continue developing block A without solving known problems. 
Operational testing verified significant block A problems, yet the Air 
Force accepted the block A system and deferred unmet performance 
requirements to block B. 

The Department of Defense recognizes the need to manage information 
systems development and acquisition by (1) defining discrete system 
development phases, (2) requiring certain activities to take place in each 
phase, (3) establishing time frames for each phase, and (4) requiring 
management reviews and approvals of each phase to ensure that sys- 
tems being acquired will satisfy mission needs in a cost effective mari- 
ner. The phases that spadoc block A proceeded through can generally be 
described as requirements setting and concept development, design, 
development, test, and acceptance. Ideally, during these phases (I) 
requirements are identified and prioritized, and concepts proposed to 
achieve the requirements are assessed to determine their risk and feasi- 
bility; (2) a specific system design is identified and assessed to assure 
that it can satisfy performance requirements; (3) the system design and 
computer programs are developed and integrated; (4) the system under- 
goes development and operational testing to assure that it meets con- 
tract specifications; and (5) the system is accepted for operational use. 
However, the Air Force permitted block A to pass through each of these 
phases without resolving critical concerns at established decision points. 
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Potential Development 
Risks Were Not 
Formally Assessed 
During Requirement 
Setting 



The requirement setting and concept development phase is supposed to 
identify and prioritize users' functional requirements and evaluate alter- 
native proposals for addressing these requirements. During this phase, 
trade-off analyses are conducted to clarify and refine the functional 
requirements and to assess the functional and technical feasibility of 
attaining them, in order to reduce program risk and costs. Techniques 
such as modeling and simulation may be used in evaluating alternative 
concepts, particularly when extensive software development may be 
required. 

The Air Force developed detailed requirements for block A, given U.S. 
Space Command's space defense mission. Two major requirements for 
block A were that the system operate in controlled mode and that it sat- 
isfy quantitative performance requirements. 

The security level required for block A had not been achieved by any 
system in 1983, when the spadoc contract was awarded. Software devel- 
opment tasks designed to achieve this form of multilevel security are 
tame-coiauming, technically demanding, and still undergoing much 
research and development Controlled mode security creates a much 
larger processing load, which slows system performance. Both Mitre and 
the Air Force recogruzed that attempting to implement controlled mode 
security would be risky and one of the two contractors compering for 
the spadoc 4 development contract stated that satisfying the security 
requirement would put the spadoc 4 acquisition at risk. However, the Air 
Force did not formally study the requirements for controlled mode oper- 
ation to determine its achievability or its impact on system performance 
before proceeding into the design phase. 



Pmhlf*m<3 Rptffln tn The purpose of Redesign phase is to translate functional requirements 

riuuieiubr>eg<ui tu and system concepts into a detailed design and to vaHdate the selected 

Surface Early m BlOCk design. Modeling and simulation techniques are used to refine require- 

A's Design Phase ments and complete the system design. The Department of Defense 



requires critical design reviews to formally review the detailed design to 
ensure that the system will satisfy performance requirements. 



As early as August 1983, only 4 months after contract award, Mitre 
raised concerns about Ford's approach toward controlled mode security, 
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Chapter 3 

Resulted in a Marginal System 



and in October 1983, questioned a performance prediction model 1 that 
Ford developed for predicting the performance of the system Pord was 
designing. Despite these indications of development problems, the Air 
Force chose to continue beyond design and into the system development 



Security Problems 
Surfaced Early 



In August 1983, Mitre pointed out that Ford's system design did not 
meet security requirements- Specifically, Ford had not included provi- 
sions in the system for storing certain levels of classified data (eg., 
unclassified, confidential, or secret). Further, in February 1984, Mitre 
stated in a letter to Electronic Systems Division that Ford's approach to 
protecting security-relevant software from unauthorized modification 
was inadequate and did not comply with the contract. In a March 1984 
letter to Electronic Systems Division, Mitre reemphasized its concerns 
about ti 
to security software. 



Validity of Performance 
Model Questioned 



The block A system specifications required Ford to provide a system 
performance model to help guide spadoc 4 design and development 
Using different potential work loads, the model was to predict how well 
alternative system designs could perform message handling, data base 
update and retrieval, computations, and other operational functions. 
During block A's design, Ford's model predicted that the proposed block 
A system design would meet all requirements, including security and 
qprs. However, as early as October 1983, Mitre told Electronic Systems 
Division that the model did not accurately reflect Ford's system. 



Significant Concerns 
Identified During Design 
Review 



A critical design review is supposed to establish the integrity of a sys- 
tem design by requiring the contractor to present a complete design and 
demonstrate that it meets performance criteria before proceeding into 
software coding. A primary product of the critical design review is spe- 
cific documentation to be used by computer programmers during soft- 
ware coding. At the conclusion of block A's critical design review in 
March 1984, over 350 design problems had been identified and changes 
were still being made to the block A design. Following this review, Mitre 
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Resulted in a Marginal System 



informed Electronic Systems Division that the risk of not completing 
block A within established time frames had increased. 

Mitre informed the Air Force following the critical design review that 
block A's risk had increased because of Ford's changing software design, 
incomplete software development controls (lack of detailed coding guid- 
ance), and an unrealistic software development schedule. Two software 
units were incomplete — one controlling the spadoc functions related to 
the 23 mission-related QPfts and the other controlling operator interac- 
tions with the system. In many cases, the design Ford presented at the 
design review differed significantly from earlier versions presented to 
the Air Force. Also, according to Mitre, Ford's proposed controls did not 
contain sufficient detail on how code should be written. This guidance 
was therefore inadequate for programmers to begin coding. Further- 
more, Ford planned to accelerate the production of code to a level 45 
percent faster than had been achieved on a previous, less complex, Ford 
software development effort, leading Mitre to question the reasonable- 
ness of Ford's software development schedule. In addition, Logicon 
reported that Ford's test documentation failed to identify specific test 
approaches, particularly for critical functional areas and the qprs. 

In all, over 350 concerns were identified at the critical design review. 
They provided early indications of system design immaturity and 
portended the development problems experienced during block A. How- 
ever, despite the deficiencies identified at the critical design review, the 
Air Force decided that the risk of postponing further development untii 
design issues could be resolved was greater than the risk of proceeding 
with an admittedly incomplete design. As a result, the Air Force allowed 
Ford to enter into the development phase, without having resolved the 
350 concerns identified during the critical design review. This began a 
pattern of deferring problem resolution that has continued to date. 



The Block A 
Development Phase 
Was Plagued With 
Technical 
Uncertainties and 
Schedule Delays 



During the development phase, the computer software programs needed 
to perform the required functions are written and integrated into the 
system design. At the conclusion of system development, component and 
system level tests are conducted to validate that system performance 
meets development specifications. 

Mitre and Logicon raised concerns regarding Ford's approach to block A 
throughout the development phase, which began in 1984. These con- 
cerns involved the quality and timeliness of software development, the 



GA0/IMTFX^9-I8 Operations Center Acquisi 



implementation of security requirements, the adequacy of Ford's per- 
formance prediction model, and block A system performance. However, 
notwithstanding clear indications during development that block A 
would not meet performance requirements, the Air Force permitted 
Ford to continue with its development and, in 1986, begin the block B 
effort 



Software Development Fell 
Behind Schedule and Was 
of Questionable Quality 



Logicon and Mitre continually raised concerns to the Air Force about the 
timeliness and quality of Ford's software development Following the 
critical design review in March 1984, Mitre predicted a 6-month delay in 
the block A schedule primarily because Ford's design was incomplete 
and still changing. By the end of September 1984, Mitre predicted the 
delay would grow to 8 to 13 months due to software development 



According to Logicon, Ford's software development documentation 
showed that many internal deadlines had been missed. Early tests were 
repeatedly postponed, and planning for software tests proceeded slower 
than expected, Logicon said. 

Logicon analyzed Ford's software periodically throughout block A 
development and expressed concerns to Electronic Systems Division 
about Ford's software quality. For example, in December 1985, Logicon 
identified 8.94 errors per thousand lines of code in one software unit, 
which Ford supposedly had tested successfully. Logicon compared this 
error rate to the 1 to 2 errors per thousand lines of code typical in analy- 
ses of other programs they had conducted. Logicon concluded that 
although Ford's quality assurance and software configuration manage- 
ment procedures were satisfactory, they were apparently not being 
enforced. As late as March 19S7, Logicon was still reporting concerns 
about the number of errors found in Ford's tested block A software. Fur- 
ther, since Logicon checked only part of the software, and Ford was 
required to correct oniy the specific errors identified, Logicon told the 
Air Force that it was likely that the completed block A software con- 
tained a large number of undetected errors, subjecting the system to a 
high risk of poor performance or inconsistent results. 

According to Logicon's July 1987 final block A report, Ford's test sched- 
ule had little built-in margin to accommodate delays in software deliv- 
eries or required retests; therefore, block A's test program posed a 
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a Marginal System 



significant schedule risk. Logieon also noted that some software devel- 
opment controls were relaxed to try to meet schedules, and that, if con- 
trols had been maintained, it is likely that the software would have had 
fewer errors. Logieon concluded that the high error rate in block A soft- 
ware led to problems integrating the software, and that it is likely that 
the block A code still contains undetected errors that could lead to simi- 
lar integration problems during block B development. 



Security ReQUirement Throughout block A development, Mitre raised concerns that Ford's 

Achievabilitv Questioned security design would not meet contractual requirements. As previously 
discussed, Mitre considered Ford's approach for isolating security soft- 
ware and for protecting security labels to be security design problems. 
Mitre was concerned that Ford's design would not achieve the controlled 
mode requirements. 

In September 1984, Air Force Space Command acknowledged the prob- 
lems Ford was having meeting the security requirements. The Command 
changed the requirement for block A to operate in controlled mode, 
instead substituting a requirement that the system operate at system 
high secret level. 2 Air Force Space Command agreed that operating the 
system at system high secret level was acceptable for block A because 
the message volume would be low enough that outgoing messages could 
be manually reviewed before release. However, the requirement for con- 
trolled mode security in block B was retained because, according to the 
Air Force, the volume of outgoing messages is expected to increase dra- 
matically under block B, making a manual security review of each 
message impractical. Although block A was no longer required to 
achieve controlled mode, the software needed to achieve this capability 
had to be retained in the block A system because it would become part 
of block B when completed. 

Ford, Mitre, and Logieon independently informed the Air Force that the 
design features implemented to achieve controlled mode security are a 
primary cause of the degraded system performance, although no one 
has measured the extent of degradation. Ford estimated the perform- 
ance degradation to be 20 to 50 percent, by analyzing a functional 
string 3 of block A code. Mitre estimated, through a similar analysis, a 



is operating at system high h 
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s much as 25 percent. According to the Mitre spadoc 
project leader, Mitre did not conduct any further analyses because it 
believed that Ford was going to undertake such a study. Ford, however, 
stated that Electronic Systems Division has never told it to assess the 
risks or effects of controlled mode.security. Therefore, the actual effect 
on system performance of continuing to strive for controlled mode 
security remains unknown. However, according to Logicon, if less strin- 
gent security requirements had been specified, it is likely that a simpler 
system design could have been developed and that this design might also 
have exhibited fewer performance problems. 



Concerns Were Raised 
About the Adequacy of 
Ford's Performance 
Prediction Model 



During block A's development, Mitre and Logicon continued to raise con- 
cerns that Ford's model was predicting inconsistent results and ques- 
tioned whether it could accurately predict performance. In March 1984, 
Mitre evaluated the model's assumptions and reported to the Air Force 
that it could not validate the model's credibility. Logicon's July 1987 
final report on block A stated that the assumptions underlying the 
model and its results have never been validated. Logicon reported that 
the model's fidelity (detail) may have been rjfficient to discriminate 
among broad design alternatives, but it produced.insufficient detail to 
accurately predict actual system performance. Logicon recommended 
that more work be done to validate the model and increase the fidelity 
by modeling system features more accurately. Even though Mitre and 
Logicon raised significant questions about the adequacy and accuracy of 
Ford's model, we found no indication that the Electronic Systems Divi- 
sion's program manager took any action to correct the model. 



Performance Problems 
Persisted Through the 
Development Phase 



Mitre informed the Air Force as early as May 1984 that block A might 
not achieve the required performance levels. However, it was not until 
summer 1986 that Mitre observed early system testing and confirmed 
that the system would have difficulty meeting the qpss. No quantitative 
measurements were collected to assess system performance during these 
tests, so Mitre proposed an evaluation plan to identify ways to improve 
system performance. 

In September 1986, Mitre and Ford began tests to identify ways to 
improve those computer processing functions that caused the system to 
perform so slowly. Mitre and Ford determined that performance could 
be improved by: (1) adding eight megabytes of system memory (increas- 
ing from 1 6 to 24 megabytes) to improve overall system processing 
speed; (2) revising software-controlling operator interactions with the 
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system and moving this from spadoc's main processor to a peripheral 
processor to improve system response to operator commands; and (3) 
improving the efficiency of software controlling access to the data base. 
Although some of these changes resulted in faster system performance, 
it was still below that required to meet the QPfis. 

Although significant development problems still had not been resolved, 
the Air Force determined thattheblockA system development tests and 
subsequent deficiency corrections had reduced the block A errors to a 
"manageable level." As a result, the Air Force decided to proceed with 
Air Force Operational Test and Evaluation Center (afotec) 1 testing in 
Air Force Space Command's test facility in June 1987. 



Operational Testing 
Further Highlighted 
Block A System 
Performance 
Shortfalls 



Operational testing of a completed system is conducted to ensure that 
the system meets the user's functional requirements and is ready for 
operational use. Tests conducted by afotec and Air Force Space Com- 
mand only confirmed what the Air Force had been told repeatedly since 
1984, that block A could not meet performance requirements, afotec 
found that block A "failed to achieve an operational capability," and 
concluded that long term efforts would be needed to achieve that 



AFOTEC Tests Identified afotec's testing was conducted to determine the readiness of block A to 
Significant Operational support operations, and was structured to address critical operational 

and Performance Problems ^^^S^^^^^.^ff^^^^f.^ 
the level of operational capability provided by block A compared to 
spadoc 3. Formal testing was conducted over a 3-day period; however, 
ahjtec continued to collect data for the next 3 months, apotec's final 
report, issued in December 1987, contained the following observations: 

• System responsiveness was not adequate; it could be overloaded during 
low levels of activity. For example, block A was unable to handle day-to- 
day message loads. As a result, it lost 10 percent of incoming message 
traffic under routine conditions, and was too slow to meet the system's 



of the system. 



n independent evaluation ol 
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System output for six of the eight astrodynamic programs 5 was not suf- 
ficiently accurate to meet user requirements. In addition, the system's 
ability to monitor and 'inform decision makers on events m space was 
hampered by poor message composition and handling, inaccurate 
message routing, and inconsistencies and ^accuracies in the data base. 
Block A did hot perform any mission critical foiidions significantly oet- 
ter than seadoc'3. For example, critical warning messages could not be 
sent to users significantly faster than spadoc3. In addition, the attempt 
to achieve controlled mode security impeded many system operations 
and did not help classify data entered into the system. 



Performance Problems 
Continued During Air 
Force Space Command 

Testing 



Beginning in January 1988, Air Force Space Command began testing 
block A in Cheyenne Mountain using the same test procedures used by 
.atotec Primarily, this testing was to verify that Fordliad corrected the 
deficiencies previously identified by akjtec. However, riot all identified 
deficiencies had been corrected and testing was suspended in February 
1988 to allow Ford to make additional connections. Mitre reported on 
some of these early test activities. 



According to Mitre reports of Air Force Space Command's test, block A 
had serious performance problems. For example, Mitre observed a con- 
tinuous 48-hour test of block A in which messages were to be received 
from but not transmitted to external agencies through Cheyenne Moun- 
tain's communications center. As a result of these tests, Mitre reported 
that the system was so slow at several points that it was virtually 
impossible to interface with it through operator consoles. For example, 
one request to retrieve a single message, which should have taken less 
than 10 seconds, took over 18 minutes to execute. Mitre also reported 
that at one point it appeared that two operator consoles were "locked 
up." In fact, the systems weren't locked; the response tookover 1 hour 
because the system was busy sorting out messages containing errors as 
well as recording system performance monitoring and evaluations called 
for under controlled mode. Mitre attributed the slow response primarily 
to (1) heavy message loads, (2) the system's inability to handle backlog- 
ged messages from the communications center, and (3) recording per- 
formance monitoring and evaluations on tape. 

Although many of these problems have been corrected over the past 2 
years and Ford continues to work on fixing others, the system's per- 
formance remains slower than required. According to Air Force Space 
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Command, the astrodynamic programs' accuracy has been corrected, the 
system's ability to handle the required message load has been improved 
but not totally corrected, and overall performance has improved but still 
does not meet requirements. Initial operational test and evaluation was 
resumed in February 1989 to verify Ford's corrections. This testing was 
completed in March 1989. Air Force Space Command plans to issue a 
final test report by May 1989. 



The Air Force 
Accepted a Marginal 
Block A System 



On the basis of system performance demonstrated in Ford's tests, the 
Air Force deferred achieving the qpbs from block A to block B and 
accepted block A on April 21, 1988. In the Ford demonstration test, 
block A met 7 of the 23 mission q?hs. The time needed to perform the 
required functions for the qphs that were not met exceeded the required 
values by as little as 5 percent to as much as 600 percent. In a March 
1988 letter to Electronic Systems Division, Air Force Space Command 
stated that, given block A test results and numerous discussions with 
Electronic Systems Division, it was clear that Ford could not meet the 
qpbs soon. Air Force Space Command made it clear, however, that defer- 
ral of the block A requirements was not intended to reduce or relax 
block B contractual performance requirements and that the specified 
qprs must be achieved in block B. 



U.S. Space Conjnand says block A is only "marginally acceptable," how- 
ever it offers a "definite improvement" over spadoc 3 in the areas of 
nuclear event threat analysis, multiple threat processing, directed 
energy threat analysis, and overall satellite data management spadoc 3 
either cannot perform these functions, or the operators must perform 
them manually. For these reasons, U.S. Space Command is prepared to 
declare the system operational at the conclusion of testing. 
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Chapter 4 



Block B's Design and Development Has 
Proceeded Much Like Block A's 



In June 1986, the Air Force modified the spadoc 4 contract with Ford to 
develop block B. The Air Force allowed block B's design and develop- 
ment to continue for more than 2 years even though (1) block B was 
built on unstable 1 block A software; (2) strong indications existed that 
the performance prediction model, while improved, was still deficient; 
(3) Ford and the Air Force remained at a standoff about whether the 
controlled mode security requirement and the Qpr requirements could 
simultaneously be achieved; and (4) Ford and the Air Force could not 
agree on whether or how to increase block B's computing capability to 
achieve the qprs. In April 1988, with 70 percent of the block B software 
written, the Air Force disapproved Ford's block B design, staring that it 
could not meet the qprs required by the contract. Ford has been revising 
the design since August 1988, and a second critical design review has 
been scheduled for April 1989. Additionally, block B software programs 
were about 80 percent complete as of August 1, 1988; Mitre estimates 
that some of this software will have to be rewritten to be compatible 
with the new design. 



Block B Development 
Was Risky Because of 
Incomplete Block A 
Software 



When Ford began developing block B software, it was still making sig- 
nificant changes to the block A software. Since the block B software 
would incorporate the ever changing block A software, the engineering 
support contractor repeatedly warned the Air Force that developing 
block B software posed serious risks. The contractor was concerned that 
block B, when completed, would include unstable block A software, 
which would make it difficult to integrate the two blocks into a total 
system. 

As early as January 1986 (before block B development began in June 
1986), Mitre identified the need for planning to assure that block B 
would riot be built on unstable block A software. Mitre informed the Air 
Force, in March 1986, that going far with the block B design before com- 
pleting block A software would constitute a risk to system design and 
development. In June 1936, Mitre again raised concerns about merging 
unstable block A software with the software to be-developed in block B. 
Mitre continued to state its concern to the Air Force in monthly project 
activity reports. Finally, as recently as March 1988, Mitre again told the 
project manager of its continuing concern about integrating block A and 
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Block B's Design and Development Has 
Proceeded Mnch Like Block A's 



block B software. Notwithstanding these persistent concerns by its engi- 
neering support contractor, the Air Force allowed Ford to continue 
developing block B, even though the block A software was unstable. 

"Rlnpk R'q PprfnrrnaTirP Ford deve *°P ed anew Performance prediction model for block B. How- 
xjiwv,^ u o x ciiuiiucuiuc everi although the model was improved, Mitre and Logicon repeatedly 
Prediction Model, Like raised concerns to the Air Force about the new model's fidelity (detail) 
the BlOCk A Model IS anA validity (reliability). While the Air Force was aware of these con- 
t\ f • j. cems with the block B performance prediction model, it continued to 

UellCient gU 0W p or d t0 X1S& ^ mo ^ to predict the relative performance of alter- 

native system designs and to make critical decisions based on data pro- 
vided by the model- 

The improvements Ford made to the model in block B increased fidelity 
somewhat. For example, to increase fidelity Ford added details such as 
the effects of operating system overhead and paging on system perform- 
ance—significant effects that were not included in the block A model. 
Ford also attempted to validate the model by comparing the actual per- 
formance of one functional string of code from block A to the perform- 
ance for that string as predicted by the model. 



However, these improvements did not overcome Mitre's and Logicons 
concerns about the reliability of the model's results. Mitre, in a February 
1988 project activity report, told the Air Force that the model was sub- 
stantially better than the block A model, but still contained a number of 
deficiencies. According to Mitre, the model did not provide enough detail 
on the operators' interaction with the system or enough detail about 
how the system accesses the data base. Further, Ford had not incorpo- 
rated changes made to block A into the block B model According to 
Mitre, these deficiencies were enough to make mode! results unreliable. 
Logicon, in its July 1987 block A final report, stated that while more 
attention was being given in block B to the performance prediction 
model, no assurances existed that its results would be substantially 
more reliable. Logicon staff who assessed the block B model maintained 
that, as with the block A model, fidelity and validation continued to be 
major concerns. Because of doubt east on the model, the Air Force still 
does not have reasonable assurance that the system, if built to the cur- 
rent design, will meet critical specifications. 
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Block B's Design and Development Has 



Disagreement Remains 
Over Whether the 
Security Requirement 
andQPRsCanBe 
Simultaneously 
Achieved 



Both the Air Force and Ford have known since July 1987 that the design 
used to achieve controlled mode security requirements slowed system 
performance, but neither party has assessed the actual extent of the 
degradation. Further, throughout block B development, the Air Force 
and Ford have not established whether a system can be built that oper- 
ates in controlled mode and meets the qprs. 

Since block B development began, Ford has maintained that it can not 
simultaneously achieve the controlled mode security requirement and 
the qprs, because in Ford's opinion, the processing rime it takes to per- 
form the security function degrades system performance by as much as 
50 percent. The Air Force, on the other hand, has maintained that a con- 
trolled mode computer system is needed to reduce the risk of security 
breaches in block B and that all Qpss are needed (with the exception of 
two that are being re-evaluated) to achieve specific spadoc mission 
requirements. The Air Force contends the contract requires Ford to pro- 
duce a system that operates in controlled mode security and meets the 
qprs. However, as discussed in chapter 3, neither Ford, Mitre, or Logicon 
has accurately demonstrated the degree to which the security design 
actually affects system performance. In the absence of this information, 
further decisions on the blockB design will, at best, be based on conflict- 
ing opinions. 



Block B Development 
Is Continuing Without 
Agreement on 
Whether Additional 
Computing Capability 
Is Needed 



As it did throughout block A, and up to critical design review in the 
blockB acquisition, the Air Force allowed development to continue 
without resolving critical issues. According to Ford, it repeatedly told 
the Air Force that it could not meet block 8 requirements without 
increased computing capability. However, block B development pro- 
ceeded without agreement between the Air Force and Ford on whether 
increased block B computing capability was needed, and if so, who 
would pay for it 

At the February 1987 block B preliminary design review, Ford pre- 
sented a design consisting of four ibm model 30S3 processors, along with 
a proposal to upgrade these machines to ibm model 3081s. The -ibm 3081 
machine contains two central processing units, as opposed to the single- 
processor ibm 3083. According to Ford, this increase from one to two 
central processors per machine would almost double the computing 
capability and substantially enhance system performance. Ford believed 
that the additional computing capacity would enable it to meet the con- 
tract requirements. Ford contends that it told Electronic Systems Divi- 
sion in February 1987 and again in April 1987 that it could not meet all 
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the spadoc contract requirements using the IBM 3083 machines, and that 
they should be upgraded to more powerful model 3081 machines. 

In May 1987, Electronic Systems Division approved Ford's preliminary 
design of four ran 3083s, and Ford continued to develop block B under 
an assumption that its proposal to upgrade the computers would be 
accepted and implemented. The Air Force was not convinced that insuf- 
ficient computing capacity was the reason that the qprs could not be 
met, nor did the Air Force agree that the contract required it to pay for 
upgraded computers. The Air Force contended that Ford had not conclu- 
sively shown that additional computing capacity would solve the spadoc 
4 performance shortfalls. The Air Force contended that Ford's system 
design was deficient, and that Ford's software approach was inefficient 
Additionally, the Air Force contended that the contract requires Ford to 
pay for any increased computing capacity needed to meet contract 
requirements- 
Ford states that it again notified Electronic Systems Division, both 
before and during the December 1987 critical design review, that it 
would not be able to meet the performance requirements without the 
upgraded computers. Accordingly, Ford's block B design presented at 
the critical design review was based on the upgraded computers (ibm 
model 3081) and on a November 1987 Ford proposal to relax the qprs by 
increasing the time specified to perform the required functions because 
Ford could not meet the original requirements. 



The Air Force 
Disapproved the 
Block B Design 



On April 1, 1988, the Air Force declared Ford's block B design unaccept- 
able and disapproved it Ford's design, presented at critical design 
review in December 1987, was based on two assumptions: (1) that the 
government would pay to upgrade the computers to bm 3081s, and (2) 
that the government would approve Ford's proposal to relax the qprs. 
Electronic Systems Division rejected both assumptions. According to 
Electronic Systems Division, Ford is responsible for delivering a system 
that meets all contractual requirements and that if upgrading the com- 
puters is required, Ford must bear the cost of the upgrade. Additionally, 
after Air Force Space Command validated all but 2 of the 23 mission 
related qprs, Electronic Systems Division disapproved Ford's request to 
relax the qprs because such a reduction would result in an operationally 
unsuitable system. The Air Force instructed Ford to propose an alterna- 
tive system design that would meet contract requirements. Ford has 
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presented an alternative design to the Air Force, which is to be 
reviewed in April 1989. Questions of whether the system will meet 
performance specifications and, if so, whether the government will 
have to pay additional costs have yet to be decided. 
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Conclusions and Recommendations 



The Air Force has spent over $235 million for a system that is now more 
than 3 years behind schedule and does not perform as required. Given 
the lack of progress that Air Force has made on spadoc to date and the 
severity of the problems that remain, we question whether the system, 
as currently designed and developed, can meet its required operational 
capability and whether the Air Force's cost and schedule estimates for 
attempting to do so are realistic. 

There are two primary causes of spadoc's problems. First, the program is 
highly complex and technically risky. Defense has put forth considera- 
ble effort in recent years to develop multilevel secure systems, yet we 
know of no mission critical controlled mode system similar to spadoc 
that has become operational. The problems in building such systems 
involve both the complexity in writing the software (the development 
tasks required are time coreuming, technically demanding, and still the 
object of active research) and the difficulty in attaining satisfactory sys- 
tem performance, given the extra processing needed to run software 
with extensive security functions built into it- This latter problem is 
especially acute when systems, such as spadoc, must perform quickly to 
provide timely warning of threat or attack. 

Second, the Air Force did not prudently manage the spadoc effort, given 
the technical difficulties and risks involved- System requirements were 
not adequately analyzed at the outset to identify which were most diffi- 
cult to satisfy and posed the greatest risk to project success, and man- 
agement strategies were not formulated and executed to accommodate 
these risks effectively. In particular, the Air Force did not formally eval- 
uate its requirements for both controlled mode security and high system 
performance to determine whether these were concurrently achievable. 
Furthermore, when problems occurred in meeting these requirements 
and questions were raised about the validity of the model being used to 
predict system performance and the adequacy of the system design, Air 
Force did not take effective corrective action- The Air Force had numer- 
ous opportunities to suspend development until problems were 
addressed and resolved, but it did not. Instead, Air Force continued com- 
mitting resources without resolving underlying technical problems, hop- 
ing that difficult, fundamental problems would somehow be resolved in 
later phases of the program. 

As a result of the technical challenges and ineffective management that 
have characterized the spadoc program, the Air Force is now faced with 
a dilemma. It has accepted and paid for a system that is only marginally 



GAO/EMTEC3918 Operations C 



d Recommendations 



useful, does not meet most contractually specified performance require- 
ments, and is not yet operational but which, according to U.S. Space 
Command, when operational will offer some functional improvement 
over the current, primarily manual system. Given that the Air Force 
now owns a system that is only marginally useful, it must decide how to 
use it cost effectively. 

Further, the Air Force must ensure that later phases of the spadoc pro- 
gram avoid the pitfalls that have hampered the effort to date. However, 
the Air Force does not appear to be doing so. Ford has been allowed to 
continue developing the block B system, even though the Air Force has 
not approved the hardware configuration, software approach, or system 
design. As with block A, continuing on this track only serves to increase 
program costs and risks with no discernable benefit 



Recommendations to 
the Secretary of 
Defense 



Due to the mission critical nature of the spadoc project, its high cost, its 
developmental difficulties, and its history of ineffective Air Force man- 
agement, we recommend that the Secretary of Defense halt block B 
development until the Air Force has complied with congressional 
requirements to submit xorad modernization programs, including 
seux)C, to Defense Acquisition Board review. During that review, the 
Secretary of the Air Force should specifically submit to the bGard: 

Hecommendations on the ultimate disposition of the spadoc block A sys- 
tem. If the Secretary recommends continuing to use block A as an 
interim improvement over the current, primarily manual system, these 
plans should include (1) an evaluation of the capabilities and deficien- 
cies of the block A system as accepted; (2) an assessment of the incre- 
mental costs and benefits of changes and modifications required to make 
the system fully operational; and (3) recommendations on how block A 
should be used based on careful analysis of costs incurred and benefits 
derived. 

Plans for the future spadoc system. These plans should include a thor- 
ough analysis of the requirements for spadoc and the feasibility of satis- 
fying those requirements, in particular controlled mode security. Plans 
should also include identification and analysis of alternative technical 
and contractual approaches to meeting the requirements, and well- 
founded estimates of costs and benefits of the alternative approaches. 
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Recommendation to 
the Congress 



We recommend that the Congress withhold further funding for the 
spadoc 4 acquisition until the Defense Acquisition Board has subn dtted 
and the Secretary of Defense has approved program plans meeting the 
objectives described in our above recommendations. 



AtfPTIPV fYimmpnt^ and The Department of Defense agreed with most of the information pre- 
n *S 1 . sented in the report but did not fully concur with several of the findings 

UUT JWaJ.Ua.tlOn and conclusions. The Department disagreed with two of the four recom- 

mendations. Defense acknowledged that the spadoc 4 program has expe- 
rienced many technical problems and uncertainties and that spadoc 4 
does not meet all performance specifications, is over cost, and behind 
schedule. However, Defense stated that the Air Force has taken the nec- 
essary actions to ensure that the later phases of the spadoc 4 program 
will avoid the pitfalls that have hampered the program to date. 
Defense's comments can be consolidated into two major areas: (1) 
whether the Air Force adequately assessed the risks of achieving the 
technical requirements, and (2) whether the Air Force effectively man- 
aged the spadoc program. 



Assessment of Potential 
Development Risks 



Defense stated in its comments that the Air Force spent 2 years develop- 
ing the requirements for the spadoc 4 system. These requirements were 
based on U.S. Space Command's need for a system to operate at multiple 
security levels (controlled mode security) and to perform certain critical 
tasks within specified time periods (stated as quantitative performance 
requirements). Defense stated that an additional 2 years and S12 million 
were invested in concept definition studies by two contractors— Martin 
Marietta Corporation and Ford Aerospace Corporation. According to 
Defense, these contractors, with technical oversight by Mitre Corpora- 
tion, developed the system requirements, assessed the risks of these 
requirements, and stated in their best and final offers that a system 
meeting both controlled mode security and quantitative performance 
requirements was achievable. Finally, Defense's comments suggest that 
because another system — the Honeywell MULTICS system — had been 
accredited and operating for several years with controlled mode secur- 
ity requirements similar to those specified for spadoc 4, that develop- 
ment risk was minimal. 

Our review of spadoc 4 source selection documentation revealed that 
meeting both the controlled mode security and quantitative performance 
requirements was not as clearly achievable as Defense's comments 
would indicate. While both contractors' best and final offers just prior to 
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contract award indicated that most spadoc 4 requirements could be 
achieved, significant concerns about the achievement of the multilevel 
security requirements raised by both contractors throughout concept 
development should have put the Air Force on notice as to the risk of 
ttiis undertaking. 

Martin Marietta made it clear in its initial security trade-off analysis 
that there had been little success in achieving controlled mode security 
and that the spadoc 4 acquisition need not be put at risk when other 
viable alternatives were available. In a subsequent design proposal, 
Martin Marietta proposed that security limitations be identified and a 
security analysis be undertaken. Further, Ford's initial design proposal 
identified hardware and software limitations, and exceptions to the 
security requirements. Because both contractors identified limitations, 
neither proposal was an unqualified endorsement that the security 
requirements could be met The initial concerns raised by both concept 
definition contractors and the limitations subsequently identified in 
Martin Marietta's later design should have put the Air Force on notice 
that an independent assessment of the achievability of the security 
requirement was needed. However, none was performed. 

Further, Defense did not provide evidence to support its statement that 
the Honeywell MULTICS system's controlled mode security require- 
ments are very similar to those specified for spadoc 4. On the contrary, 
our analysis of the Honeywell MULTICS system shows that its secure 
operating capabilities are not similar to those capabilities required for 
SPadoc 4. MULTICS is a stand alone time-shared system with no elec- 
tronic connection to other computer systems, and thus, all classified 
information is protected within the system. However, because spadoc is 
connected to other computer systems (some with lower levels of security 
than spadoc) it must be able to prevent compromise of classified data. 
Further, MULTICS is not a real-time system. Therefore MULTICS pro- 
vides a security capability but does not have to simultaneously meet 
stringent performance requirements. Meeting both performance and 
stringent security requirements has been a major technical problem in 
spadoc, a problem that MULTICS does not have to overcome. Because of 
these significant differences between the Honeywell MULTICS and the 
spadoc 4 systems, the Air Force's comparison is not valid. 

In commenting on the draft, report, Defense stated that "the questions 
about the ability to achieve both controlled mode security and the qprs, 
as well as the computer capacity needed, have been resolved (underscor- 
ing supplied)," The Air Force claims to have identified a hardware 
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architecture, using ibm model 3090 computers, that not only will meet all 
block B requirements, including quantitative performance and con- 
trolled mode security, but will also possess the growth capabilities 
needed to meet block C requirements. However, Defense stated that the 
Air Force "is in the process (underscoring supplied) of developing a 
block C Requirements Specification, to include block B's final design and 
performance capability." It is unclear to us how a determination can be 
made that using an ibm 3090 architecture will be the most efficient and 
cost effective approach to meet block C requirements when these 
requirements have not been developed. Therefore, after receiving 
Defense's comments, we asked Electronic Systems Division to explain 
their rationale for concluding that the ibm 3090 based architecture will 
meet all block B requirements and have the growth capabilities needed 
to meet block C requirements. 

We were told that this rationale is based on Ford's modeling of estimated 
block C requirements using an improved version of the block B perform- 
ance prediction model. However, we note that while Ford's model for 
predicting system performance has been improved, it has not been vali- 
dated and problems with it are stall being resolved. Further, the Elec- 
tronic Systems Division has not performed sufficient analyses to 
conclude that this is the most cost effective and efficient means of 
achieving spadoc requirements. 

Air Force Management of Defense disagreed with our assessment that the Air Force did not pru- 
the SPADOC Program dently manage the spadoc effort given the technical difficulties and risks 

involved. While we reported that the Air Force repeatedly deferred 
problem resolution to later phases of the acquisition effort, Defense 
characterized the Air Force's efforts as "positive actions to manage a 
very complex major acquisition...." Three examples are illustrative of 
the difference of opinion. 

First, we reported that while the Air Force identified over 350 action 
items at the block A critical design review, the Air Force decided to 
allow the contractor to continue developing the design without first 
resolving the problems. We noted that following the critical design 
review, Mitre characterized Ford's design as being in a state of flux 
because the design baseline had not been stabilized, software engineer- 
ing controls were not developed, and the software coding and testing 
effort was overly ambitious. 
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Defense stated that the Air Force did not allow the action items to go 
unresolved; instead, it required Ford to address and correct the 350 
action items, an effort that Defense states was completed and approved 
by the Air Force in July 1984. We did not find this to be the case. In 
October 1984, Mitre again observed that significant portions of the 
design had still not stabilized, the test program was still being defined, 
and the software development effort was unlikely to succeed. In fact, 
the block A software design was not stable when the system was 
accepted in April 1988 and Ford's test program and software coding 
effort became a major source of problems that continued to delay pro- 
gram development While the Air Force states it has managed the pro- 
gram prudently and resolved the action items, we found that Air Force 
management actions taken to date have not resolved identified 



Second, we reported that Mitre and Logicon continuously raised con- 
cerns about problems Ford was having meeting block A's security 
requirements. As early as August 1983, Mitre pointed out that Ford's 
design did not meet security requirements, and in February and March 
1984, Mitre stated that Ford's approach to protecting security-relevant 
software from unauthorized modification did not'comply with the 
contract. 

Defense agreed that Mitre raised several issues concerning Ford's secur- 
ity design; however, Defense stated that we failed to recognize actions 
taken by the Air Force in response to these concerns. Defense stated 
that the Air Force made the contractor respond to each issue raised by 
Mitre and correct its design accordingly. However, wedid not find evi- 
dence that the Air Force's actions resolved the problems, hi fact, in Sep- 
tember 1984, the Air Force recognized that the security requirements 
could not be achieved quickly and therefore, decided to defer achieving 
them to block B. 

However, action to defer achieving the requirement did not solve the 
problem either. Throughout block B development, Ford maintained that 
it could not simultaneously achieve the security and quantitative per- 
formance requirements, and continuously sought relief. Because Ford's 
block B design presented at critical design review could not meet all con- 
tract requirements, the Air Force rejected the design. As a result, the Air 
Force and Ford have spent the last year trying to identify a design to 
simultaneously achieve the security and quantitative performance 
requirements. While Tefense maintains that the Air Force did not allow 
the contractor to continue development without resolving problems, 4 
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Third, we reported that in April 1988 the Air Force acknowledged its 
inability to meet the block A qprs without a major redesign of the block 
A system. Thus, the Air Force deferred achieving block A quantitative 
performance requirements to block B and accepted block A. According 
to the Vice Commander, Electronics Systems Division, this was done "to 
get some rnirumal and marginal capability into Air Force Space Com- 
mand's hands." 

However, in the same month, the Air Force rejected Ford's block B 
design. According to Defense, Ford's efort, which included a major 
redesign needed to correct block A's deficiencies, also could not achieve 
the mission QPRs. As a result, the Air Force now owned block A, which 
did not meet its needs, and had block B on hold because it too could not 
satisfy those needs. The Air Force now plans to upgrade the spadoc sys- 
tem's computers to faster, more powerful IBM model 3090s in another 
attempt to achieve controlled mode security and the quantitative per- 
formance requirements. This computer upgrade is intended to replace 
the computers that the Air Force bought when it accepted block A. How- 
ever, as we noted earlier, the Air Force has not performed the analysis 
necessary to assure itself that this second set of computers will satisfy 
spadoc 4 requirements. 

The Department of Defense believes that the Air Force took positive 
actions in attempts to overcome what it calls "the root causes" of the 
SpadOC 4 program's problems. According to Defense, the SPadOC 4 con- 
tractor too rigorously implemented the security requirements, inef- 
ficiently designed the software, poorly integrated the software, aid 
ineffectively managed its subcontractors. Notwithstanding Defense's 
claims, it does not alter the fact that Air Force management allowed the 
situation to go on for many years before initiating action to resolve 
problems. As we pointed out in our report, the Air Force had numerous 
opportunities to suspend development until problems were addressed 
and resolved, but it did not. It was not until April 1988, when the Air 
Force rejected Ford's block B design proposal, that the Air Force began 
to take positive action to halt what it called "the root causes" of spadoc 
4's program problems. 

Finally, Defense disagreed with our recommendations to withhold fund- 
ing and halt spadoc 4 development. Defense stated that halting block B 
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would only serve to exacerbate the cost and schedule problems previ- 
ously experienced and would result in an unnecessary delay in the criti- 
cal operational capabilities. Defense's comments provide no evidence to 
support its claim that halting development would exacerbate cost and 
schedule problems. In fact, continuing to develop a system with an 
unstable design, such as a constantly changing hardware architecture, 
generally results in the need for later changes, as well as cost growth 
and schedule delays. Our recommendations are designed to minimize 
future cost growth and schedule delays. Halting development will give 
the Air Force an opportunity to reassess its critical requirements and 
resolve the complex technical issues facing the spadoc program. First; 
U-S. Space Command must determine whether controlled mode security 
is absolutely necessary to achieve its mission or whether other alterna- 
tives are viable. Second, if controlled mode security is found to be neces- 
sary, the Air Force must determine whether this requirement is 
achievable without undue cost, schedule, and performance impact 
Third, before approving an upgrade to the system's computers, the Air 
Force must determine that the model being used to predict system per- 
formance is capable of accurately predicting the performance of the new 
computers. 
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Current ITW/AA System 



Cheyenne Mountain 



Space 



Miiftary and Civilian Leaders 



"HXr" 
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Appendix II 



Modernized ITW/AA System 




Cheyenne Mountain 



Defense 
Operations 




Communications 
System Segment 
Replacement 




Gisnite 
Sentry 








Command 

Processing 
and Display 

System 
Reptacemeftf 






t 




I SCIS 1 



Military and Civilian Leaders 
' SCIS Survivable Communications Integration System 
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Comments From the Department of Defense 




Information Management and 
Technology Division 



This is the Department of Defense (DoD) response to the 
General Accounting Office (GAO) Draft Report, "SPACE DEFENSE: 
Management an£ Technical Problems Delay Operations Center 
Acquisition," dated January 6, 1989 (GAO code 510275, OSD Case 
7872). The DoD partially agrees with the report. 

Although many of the facts identified in the draft report 
are correct, the Department of Defense does not fully concur with 
several of the findings and conclusions. Therefore, several 
points of clarification are provided. The Cheyeime 1 



' for space defensive agencii 
iriginally planned, the 



i unfortunately. 



: and behind 
schedule. The SPADOG 4 requirements established by the Air J 
were the results of four years of effort and reflect what thi 
Space Command definitely needs in order to perform its space 

raanage a very complex major acquisition which is critical to 



'. 4 program avoid I 



:ombine the Cheyenne I 
program element and is preparing for a Defense Acquisition Board 
review, now anticipated in June 1989. It continues to be the DoD 
position that the SPADOC 4 program is still the most prudent 
approach for extending the space defense and surveillance mis- 
sions into the 21st Century. 
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appreciates the opportunity i 
""- ""-- '^tailed DoD comrai 

enclosed. Addition, 
ately provided to oembers of yi 



<^2CH 



Gordon a. Smith 
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From the Department of Defense 



DEPARTMENT OF DEFENSE COMMENTS 



: Formally Assessed. 



ingle center 
t data process 
! GAO explained that the SPADOC 4 : 



; being developed : 



not formally assessed during ( 

3 and 
the GAO found that 

requirements for block A, based on the D.S. Space Cotamand 
space defense mission, with oajot requirements that the 
system operate in controlled mode and that it satisfy the 
quantitative performance requirements. The GAO pointed c ' 



POD RESPONSE; Partially < 
facts are correct, the Dol 



; regarding the SPADOC • 



true that SPADOC 4 is the fin; 

to develop and modernize a single center for space defense 

agencies, and unfortunately, as originally planned. 



of the SPADOC 4 requirements and did not attempt to develop ■ 
system that was beyond the state-of-the-art. 
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.ts space defense i 
'as the need for a 
fitliin specified time periods" in order for the SPADOC I 



! adequate 
operators. These time critical requirement; 
quantitative performance requirements, or Ql 
Additionally, because of the different secus 
involved, as well as the time constraints, l 
system also had to be capable of operating ; 
security levels (controlled mode security). 

During 1982 and 1983, the Air Force spent oi 
formal, competitive Concept Definition ohasi 
SPADOC 4 -■■-- - 
Aerospace Corpora t: 

requirements, formally assess the risics of these requirements 



proposals for developing the SPADOC '__ 
Contrary to the GAO report, the Sest and Pinal Offers of both 
contractors stated that a system meeting both controlled mode 
security and quantitative performance requii 



were approved by the Air Force before 
awarded, over the course of the block 
opaents, these requirements 

At the time of contract award, the Honeywell Hultics system 
had already been accredited and operating for seven years 
with controlled code security requirements very similar to 
those specified for the SPADOC 4.* Today, the Honeywell 
Secure Coaaunications Processor and the SACDIN program have 
also been accredited as meeting security requirements 
significantly more stringent than those specified for the 
SPADOC 4. Still, during Concept Development, the Air Force 
intentionally selected only the ninimum requirements for the 
SPADOC 4 because it would have been unacceptably costly, tinw 



isumingj and i 

; full set of security features, 

the SPADOC 4 requirements established by the i 



space defense mission. These requirements 
assessed by three separate contractors, > 
Force, and were found to be technically i 
concert with previously developed system: 
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Force took positive action in limiting the scope of the 
security requirements in order to minimize potential 
development risks. 



found that, although block A has been in development 

*""3 required missior 
tually specified 



perform 14 of 23 required mi: 



i addition, the ( 



■ *983, 
functions quickly enough to satisfy 
quantitative performance requirement 
found that the systea does not ensu: 
secret information will not be sent to una 
owners and operators. The GAO noted that, from the 
beginning, the Air Force was aware that achieving some block 
A requirements would be rjsKy and needed close management 
oversight. According to the GAG, Mitre repeatedly told the 



: security requi resents, 
i early 1984. The t"~ 



out that the Ford system did 

and then reenphasized its co: 

also reported that, as early 

Air Force that the perfonaam 

reflect the Ford system. In addition, the GAO found thai 

March 1984, at the conclusion of the block A critical desi< 

review, over 350 design problems had been identified and 

changes to the block A design were still being made- She ( 



though requi: 
problem i 



incomplete design < 



xtially Concur. While problems did surface 
1 development, the DoD 



Slock A develoj 
in April 1988, 



i April 1983 and as the system 

14 of the 23 originally 

r, as noted i.n the GAO report. 
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reviewed and automatically tagged as SECRET. All recipients 
of block A messages are authorized to receive SECRET 
information. Therefore, classified information will not be 



i unauthorized i 



After dev< 
several issue: 
and inadequaci 



icurity design 
ts model. However, the GAO failed to 
: taken by the Air Force in response to 
i respect to the quality of design. 



1 earlier versions presented I 
Additionally, by the tine the block A model inadequacies were 
veri :ied by the Air Force, block A development had progressed 
far enough to begin naking actual perfo 
The block A model was then deleted as a system perfori 
prediction tool, and only used as a tool to identify 



in-stressed i 



w (CDS) action items tc 

o continue developing tne system, in: 
n items were resolved and closed through 

npufc, approved block A CDS on July 24, 19S4 



cal Design 
lloving the 
tead, all 



i,„ And Schedule Delays. The GAO found 

that, throughout the development phase of block A, beginning 
in 1984, Mitre and Logicon raised concerns regarding the 
approach of Ford. According to the GAO, these concerns 
involved the quality and timeliness of software development, 
the implementation of security requirements, the adequacy of 
i performance prediction model, and block A system 
The GAO found that the Air Force allowed Ford 
developing block A without solving known 
iblems,. despite expresssed concerns and identified 
ibleras. The GAO pointed out that, at any point in the 
icess, the Air Force could have suspended development until 
> problems were addressed and resolved, but it did not do 
The GAO concluded that, as a result, the Air Force did 
: prudently manage the SPADOC effort, (p. 4, 
19, p. 25, pp. 31-38, p. 53/GAO Draft Report) 



; and schedule delays and 
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that it should have suspended formal development, 
therefore did not prudently manage the SPADOC effort. 

As stated in the DoP response to Finding 8, the aic Force 
directed the contractor to incorporate technical issues 
raised by Mitre and did not approve CDR until all 350 action 
items were closed. After the CDR, as Mitre and Logicon 

> performance, the 
Lqht of Ford, to 

required 
regulations. For example; 

1984, the Air Force directed Ford to publish a 
Technical Report and Software Integration Plan to 
better manage the database and software development process, 
and initiated monthly Software Status Reviews. 

- In January 1985, the Air Force initiated management of 
Ford's year long software integration process through 
establishment of weekly milestone reports. 

- In September 1986, Mitre (representing the Air Force) an 
Ford began tests to identify ways to improve those computer 
processing functions that caused the system to perform so 



Although formally suspending block i 

System's Division and Air Force Space Command (AFSPACECOM) , 
nor the DSSFACECOM, feel that it would be in the best 
interest of the Government to do so. Such action would hi 
only resulted in a loss of trained contractor staff, thus 
increasing both costs and risk, while further delaying 
milestones. Both the Air Force and the OSSPACECOM believi 
appears to be the prudent 
ip^bilities needed for the 
:e defense mission (See also 



although significant development 
resolved, the Air Force determine 
development tests and subsequent 
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te Department of Defense 



with Air Force Operational Tes 
(AFOTEC) testing in June 1987. 
AFOTEC testing 
of block A to support operati 
address critical operational 
the AFOTEC testing confirmed i 



i "manageable level." The Gi 
s Air Force decided to proce< 

According to the GAO, the 



: Force had been told 



repeatedly since 1984, that block A could i 

performance requirements. According to the GAO, the AFOTEC 

found that block A failed to achieve operational capability 



i January 1988, the J 



previously identified. Although a final test report has not 
yet been issued, the GAO found that indications are that 
block A performance problems have continued, {p. 4, pp. 38- 
41/gao Draft aeport) 

; that operational 



:esting further highlighted block A system perfoi 
i be recognized. 



i significant number of the AFOTEC t 



i take advantage of 



findings would provide early feedback 



Facility (SOSTF) 
irly feedback on c 
All the disCrepan 



block A bears 



; Accepted A Marginal Block A System. 



■Rs). According to the GAO, in March 1988, i 

ice Command stated that, given the block A test results, ii 
; clear that Ford could not meet the QPRs anytime soon. 

reported that, in April 1988, the Air Force decided 



) defer meeting the block A perron 
:hieving' controllec" 
! block A system l 



>n trolled mode security to block B, and accepted 
system to "get some minimal and margi 
capability into Air Force Space Command's hands." 



requirements and 

and marginal 
Is." in 
accepted, 295 



According to the GAO, Ford f 



fied as significant and that 
>re the system could become operational, 
plans have been made to address the 
i because they are not considered to be 
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contract requirements. According to the GAO, testing to 
verify that the 90 deficiencies have been corrected began in 
September 1988, and the Air Force anticipated declaring block 
A operational in January 1989. (pp. 3-5, pp. 19-20, pp, 42- 
43/GAO Draft Report) 



t number of performance requirements 



: block 



(IOC) critical servit 
definite improvements ( 



i the "marginal" category. Block A will provide 
immediate benefits in the areas where SPADOC 3 is deficient 
today: nuclear event processing, data management, message 



to block B. Controlled mode security had already been 
deferred to block B in 1984, not April 1988. The remaining 
two QPR's have been fixed as part of the 90 open deficiencies 
that the contractor agreed to fix before the system goes 



> address I 



: block i 



urect that no plans were made to address the 20! 
fact/ it was determined that the 
s the most prudent way i 
m. With the acceptance 
' legally develop i " 
under the block A maintenance contract, 
significant operational impact either have, or are currently 
being incorporated in the block A system through the Materi; 
improvement Program Review Board. Those enhancements with 
less operational impact will be analyzed, prioritized, and 
implemented as required by the block A Software Configuratii 



Control Sub-Board as normal post-IOC ' 



releases. 



developing block 5 in J: 
block A problems could : 



; GAO reported that (1) block l 



(3) Ford and the l 



trolled mode ; 
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requirements could simultaneously be ; 

QPRs. The GAO discussed" concerns" raised" and efforts made t 
overcome these problems, both before and after the dune 1986 
decision to begin block B development. The GAO found that, 
despite these problems, the Air Force allowed the block B 
design and development to continue for more than two years. 
The GAO reported that, in April 1988, after nearly two years 
of development and* with 70 percent of the block B software 
written, the Air ~ ' ■ - -■ ~ - '- ■ - - ' 
proposal because 

QPRs, The GAO recognizee tnat, at tne time, tne wt Forc« 
emphasized to Ford that it is required to deliver a systei 
design that will meet all operational requirements. The ( 
reported that, in August 1988, Ford agreed to redesign block 
". operational 
; scheduled : 
-ised proposal. The GAO 
points out, however, that the questions of whether the system 
will meet performance specifications and, if so, whether the 
Government will have to pay additional costs, have yet to be 
decided. The GAO concluded that, as with block A, continuing 



» Draft Report) 



tp, 5, pp. 20-21, pp. 



described in the above finding : 

) concli 
i block 3. Instead, the * 



i block B development began in June 1986, 



i upon what proved I 



l changing block i 



Logicon raised issues along these lines. Therefore, i 
■ presented its block B design at the CDS, tl 
! disapproved it, due in part to the problems sti' 
:ing in block A. As mentioned in the GAO finding, 
ractor' has agreed to redesign block B, now based < 
.e block A baseline, and will present this design 
i CDR, now scheduled for April 1989. Additionally, 



: should also be i 



! block B code already developed, 
i 10 percent will need to be 



tested or accepted. 
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With respect to the GAO statements concerning the inadequacy 
of the block B model, the Air Force has expended substantial 
effort in evaluating and analyzing the block B performance 



: modeling effort. Considerable effort was spent analyzing 
the underlying assumptions of the model, and identifying 



and conducted t 



problems that were identii 
been fixed, or are continuing to 
is aware of what these problems < 
must be resolved prior to the bli 



capacity needed, have been resolved, since April 1988, the 
Air Force has undertaken a very aggressive effort to process 
engineering change proposals that will upgrade the hardware 
architecture, improve software design and add system 
functionality so that a successful CDH can be achieved, for 
example. Air Force resource: 
hardware architecture that i 
requirements, including quai 
controlled mode, but will a) _ 

: block C requirements. This 



FIHDIHG G: Slock C Is Planned, 



capability needed to manage the U.S. Space Command < 
defense mission. According to the GAO, block C is to 
maintain and update orbital data and make orbital position 
predictions on at least 10,000 potential objects in the data 



: GAO reported that, 
: operational by ."""" 
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been reached. Additionally, the i 



; complying with 



Congress anticipated by June 1989. 

~ -"■-■- it to award block C 

: Requirements 
Specification, to include block 3's final design and 
performance capability. Additionally, the Air Force is 
purposefully keeping its acquisition strategy for block C 
open to incorporate any future lessons learned from block B. 

FIHDIKG Hi SPflDOC Acquisition Costs Have Risen And 



i Kay 1982, "to $437 million, as cf Marco 1988. 
i reported that, in addition, the scheduled completion 
: three blocks of SPADOC 4 has been delayed significantly, 
:h block C now projected f " 



: 1994, rather 



.ly planned, 
iginal costs dl<: 
expenditure thresholds that 



although 
expend! ti 
oversight throughout 



t required high level 
OSD and the Air Force 

though t 



this original < 
«e these thresholds early in the life < 
The GAO pointed out that, in 
conjunction with the direction to consolidate all NOHAD 
modernization programs, including the SPADOC, under Defense 
Acquisition Board oversight, the Congress also required that, 

Dlidated program 

! progress the 

SPADOC to date and the severity of the problems that reizain, 
the GAO questioned whether the systea, as currently designed 
and developed, can meet Its required operational capability, 
and whether the Air Force cost and schedule estimates for 
attenuating to do so are realistic, (o. 3, dd- 22-24, p. 
52/GAO Draft Report) 

POD RESPOMSB: Partially concur. The SPADOC 4 costs have 
risen and block C is now planned for completion in FY 1994. 
It should be recognized, however, that through proper 
management attention, the Air Force has taken action to 
address the SPADOC problems and has accepted the schedule 
delays to better ensure that the system will meet 
requirements. The Air Force is complying with all 



> response to Findings E and 
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a From the Department of Defense 



F, substantial progress has been made to both blocks A and B 
since the GAG drafted this report. Therefore/ the DoD is 
confident that the SPADOC 4 will inset its required capability 

be presented to the DAB revii 

FIHDIHG I; Future Prospects 



: realistic. 



program t 



: Force to prudently manage the SPADOC 



The GRO stated that the Air Force is now faced with a 
dilemaa — it now owns a systea that is only marginally useful 
and it now oust decide how to use the systea cost 



pitfalls that have hampered t 

concluded , however, 

doing so. The GAO pointed out that Ford has been allowed l 

continue developing the block B systea, even though the Air 

Force has not approved the hardware configuration, software 

approach, or systea design. The GAO concluded that 

; track only serves to increase prograa 



i prudently manage 
capability 



demonstrated by 1 



Additionally, 



:eas were di 
! the failui 



i findings, the 
<de security had bsen 
; systea for seven years 



implementation of security features, inefficient software 
design, poor integration, and lack of subcontractor 
management by the SPADOC 4 contractor. These areas are being 
closely monitored by the Aii 
improvements have been seen 

offer definite improvements over existing SPADOC 3 
capabilities, and the switch to ISM 3090's provides < 
architecture that will handle both controlled mode si 
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' block C requirements. 

i the SI 
actor i 
has recently been achieved, the DoO believes t 
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Comments From the Department of Defense 



RECOMMENDATIONS 



complied with congressional requii 



(p. 5, p. 54/GAO Draft Heport) 



into a single program element and a DAB review is anticipated 
for June 1989. as discussed in the DoD responses to findings 
E, F and G, the DoD is confident that the Air Forces has 
taken the actions necessary to ensure the success of the 
SPADOC program* Halting block B now would only serve to 
exacerbate the cost and schedule problems previously 
experienced and would result in unnecsessary delay in the 



:ical 



capabilil 



I specifically submit 1 



stary recommends continuing to use block I 
improvement over the current, primarily manual systen r these 
plans should include (1) an evaluation of the capabilities 



_ lired to make the sysfcea useful 
operationally, and {3} recoeaendations on how block a should 
be used, based on careful analysis of costs incurred and 
benefits derived, (pp. 5-6, pp. 54-55/GAO Draft Report] 

rorce is preparing for a June 
; Congress. The Secretary of 
;sues required by the DAB and 



1989 DAB review and report t 
the Air Force will address i 
congressional direction. 



Force should specifically submit 
future SPADOC system. The GAO ft 
plans should include (1) a thorough analysi 



l and analysis of altern; 



> technical and 
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Comments From the Department of Defense 



Now on pp. 4-5 and p. 34, 



Now on p. 5 and p. 35. 



contractual approaches to meeting the requirements, and (3) 
well founded estimates of costs and benefits of the 
alternative approaches, (pp. 5-6, pp. 54-55/GRO Draft 
Report) 

POD RESPONSE : Concur. See the DoD response to 
Recommendation 2. 

> asCQKMKHDATlOH 4? The GAO reconnended that the Congress 
withhold funding for the SPADOC 4 acquisition until the 
Defense Acquisition Board has submitted, and the Secretary of 
Defense has approved, program plans meeting the objectives 
" ~"T recc==endations. (p. 6, o. 
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